Game apparatus having general-purpose remote control function

ABSTRACT

A game apparatus is capable of controlling infrared radiation emitters for radiating infrared light. The game apparatus executes game processing using imaging information on the infrared light obtained from an input device. The input device takes an image of the infrared light radiated by the infrared radiation emitters, and also receives an input from a user. The game apparatus comprises pattern storage means; selection means; and radiation control means. The pattern storage means stores at least one signal pattern of an infrared signal usable by a control target device. The selection means selects at least one of the at least one signal pattern stored by the pattern storage means, using the input received by the input device. The radiation control means causes the infrared radiation emitters to output an infrared signal of the signal pattern selected by the selection means.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/020,872, filed Feb. 4, 2011, now allowed, which is a continuation ofU.S. application Ser. No. 11/598,787, filed Nov. 14, 2006, now U.S. Pat.No. 7,905,782, which claims the benefit of the disclosure of JapanesePatent Application No. 2006-211183, filed on Aug. 2, 2006, each of whichare hereby incorporated by reference in their entirety.

BACKGROUND

1. Field of the Technology

The present technology relates to a game apparatus having ageneral-purpose remote control function, and more particularly to a gameapparatus having a general-purpose remote control function, which iscapable of remotely controlling a remote-controllable device for generalpurposes.

2. Description of the Background Art

Conventionally, an exemplary technology for an apparatus having ageneral-purpose remote control function is disclosed in patent document1 (Japanese Laid-Open Patent Publication No. 2-198299). Patent document1 discloses a remote control transmitter capable of remotely controllingdifferent types of devices produced by different manufacturers. Thisremote control transmitter reads, from a memory which stores data on thedifferent types of devices produced by different manufacturers, data ona specific type of device, and thus remotely controls the device.

The remote control transmitter disclosed in patent document 1 merely hasa function of remotely controlling the different types of devicesproduced by different manufacturers. Such a remote control transmitteris not usable for a game apparatus, in which means for transmitting datafor remote control is also used for other purposes. Therefore, in orderto control a remote-controllable device while a game is being played,the player needs to leave his/her hand or the like from the input devicefor the game, and picks up a remote control transmitter for the device.This is very inefficient.

SUMMARY

Therefore, a feature of the present technology is to provide a gameapparatus including an input device which is capable of operating adevice other than the game apparatus.

The present technology has the following features to attain the above.The reference numerals, additional descriptions and the like inparentheses in this section of the specification indicate thecorrespondence with the embodiments described later for easierunderstanding of the present technology, and do not limit the presenttechnology in any way.

According to a first aspect is directed to a game apparatus (3) forexecuting game processing. The game apparatus is capable of controllinginfrared radiation means (markers 6R and 6L) for radiating infraredlight. The game apparatus executes the game processing using imaginginformation (marker coordinate sets) on the infrared light obtained froman input device (controller 5). The input device takes an image of theinfrared light radiated by the infrared radiation means and receives aninput from a user. The game apparatus comprises pattern storage means(flash memory 18), selection means (CPU 10 or the like for executingstep S11, S44, S57 or S79; hereinafter, only the step number of thecorresponding processing will be described in this section of thespecification), and radiation control means (S12, S45, S58 or S80). Thepattern storage means stores at least one signal pattern of an infraredsignal usable by a control target device. The selection means selects atleast one of the at least one signal pattern stored by the patternstorage means. The radiation control means causes the infrared radiationmeans to output an infrared signal of the signal pattern selected by theselection means.

According to a second aspect, the signal pattern may be selected usingthe input received by the input device.

According to a third aspect, the infrared signal may represent at leastone of an image and a sound to be reproduced by the control targetdevice.

According to a fourth aspect, the infrared signal may represent aninstruction for causing the control target device to perform apredetermined motion.

According to a fifth aspect, the control target device may be a displaydevice (TV 2) connected to the game apparatus for displaying a gameimage obtained as a result of the game processing. When a predeterminedbutton provided in an input device is operated, the selection meansselects a signal pattern for turning on the display device.

According to a sixth aspect, the game apparatus may further comprisemotion determination means (S61) for determining a motion of a characterappearing in a game space generated by the game processing, using theinput used for selecting the signal pattern.

According to a seventh aspect, the game apparatus may further comprisedisplay control means (S46) for causing a display device to display animage representing the input used for selecting the signal pattern.

According to an eighth aspect, the game apparatus may further compriseinput information storage means (S43) and game data determination means(S48). The input information storage means stores input history data(635) representing a history of inputs used for selecting the signalpattern. The game data determination means determines game data (racecourse data 636) to be used for the game processing based on the inputhistory data.

According to a ninth aspect, the game apparatus may further comprisemotion determination means (S61) for determining a motion of a characterappearing in a game space, using the input received by the input device.At this point, the selection means selects a signal pattern using theinput used for determining the motion of the character.

According to a tenth aspect, the game apparatus may further compriseinput information storage means (S53) for storing input history data(635) representing a history of inputs used for determining the motionof the character. At this point, the selection means selects a signalpattern based on the input history data.

According to an eleventh aspect, the game apparatus may further comprisegame processing means (S61) for executing the game processing using theimaging information on the infrared light radiated by the infraredradiation means.

According to a twelfth aspect, the game apparatus may further comprisemode switching means (S3, S8) and game processing means (S9). The modeswitching means switches a device operation mode and a game processingmode to each other at a predetermined timing. The game processing meansexecutes the game processing using the imaging information only in thegame processing mode. At this point, the selection means selects asignal pattern only in the device operation mode. The radiation controlmeans continuously causes the infrared radiation means to output theinfrared light in the game processing mode.

According to a thirteenth aspect, in the device operation mode, a partof a plurality of buttons provided in the input device may be eachassociated with a signal pattern to be selected by the selection meanswhen the respective button is operated. At this point, the modeswitching means switches the device operation mode to the gameprocessing mode under at least one of the condition that there has beenno input to any button provided in the input device for a predeterminedtime duration and the condition that a button which is not associatedwith any signal pattern has been operated.

The present technology may be provided as a storage medium having storedthereon a game program which is executable by a computer of a gameapparatus to realize each means of the game apparatus.

A fourteenth aspect is directed to a game system including an inputdevice (controller 5), having a plurality of types of operation keys anda wireless communication function, and a game apparatus (3) wirelesslycommunicable with the input device. The game system comprises aninfrared radiation device (markers 6R and 6L) connected to the gameapparatus. The game apparatus includes a memory (flash memory 18) andinfrared control means (CPU 10, etc.). The memory stores at least onesignal pattern of infrared light for controlling a control targetdevice. The infrared control means, in accordance with key informationcorresponding to an operation key which is transmitted from the inputdevice by wireless communication, reads a signal pattern correspondingto the key information from the memory and controls the infraredradiation device to output the infrared light based on the signalpattern.

According to a fifteenth aspect, the input device may include an imagingcamera and position information transmission means. The imaging cameratakes an image of the infrared light which is output from the infraredradiation device. The position information transmission means transmitsposition information on the position of the infrared light on an imagetaken by the imaging camera to the game apparatus. In this case, thegame apparatus includes game processing execution means (S61) forexecuting game processing based on the position information and the keyinformation received from the input device. The infrared control meansreads a signal pattern from the memory in accordance with the state ofthe game processing, and controls the infrared radiation device tooutput the infrared light based on the signal pattern.

According to the first aspect, the imaging information on the infraredlight which is output from the infrared radiation means is obtained bythe input device, and this imaging information is used for the gameprocessing. Therefore, the player can execute the game processing byoperating the input device. In addition, the radiation control means cantransmit various data to the control target device by radiating aninfrared signal of a signal pattern selected by the selection means.According to the first aspect, a game apparatus having a general-purposeremote control function capable of operating another device in additionto executing the game processing using the input device can be provided.

According to the second aspect, the signal pattern of the infraredsignal to be output from the infrared radiation means is selected by aninput to the input device. Therefore, the player can transmit variousdata to the control target device by operating the input device.

According to the third aspect, the game apparatus can cause the controltarget device to display a desired image or output a desired sound bytransmitting data on the image or the sound to the control targetdevice.

According to the fourth aspect, the game apparatus can cause the controltarget device to perform a desired motion by transmitting such aninstruction to the control target device.

According to the fifth aspect, the display device for displaying thegame image can be turned on using the input device. Therefore, theplayer only needs to operate the input device to start the game. Thetrouble of operating a plurality of remote controls is eliminated.

According to the sixth aspect, a motion of a character in the virtualgame world can be associated with a motion of a control target device inthe real world. This makes the game more amusing.

According to the seventh aspect, the player can visually check theoperation particulars performed to control the control target device.This makes it easier to operate the control target device.

According to the eighth aspect, a setting regarding the game in thevirtual game world can be associated with a motion of the control targetdevice in the real world. This makes the game more amusing.

According to the ninth aspect, a motion of the control target device inthe real world can be associated with a motion of a character in thevirtual game world. This makes the game more amusing.

According to the tenth aspect, a motion of a character in the game worldcan be reproduced by the control target device in the real world.

According to the eleventh aspect, even while the game processing isbeing executed using the imaging information, another device can beoperated using the input device.

According to the twelfth aspect, when the game processing is executedusing the imaging information obtained by taking an image of theinfrared radiation means using the input device, the infrared radiationmeans does not output any infrared signal and continuously outputsinfrared light. The input device can obtain accurate imaging informationby taking an image of the infrared radiation means in a secure state.Therefore, the game processing can be executed more accurately.

According to the thirteenth aspect, the game apparatus determineswhether or not to control the control target device and automaticallychanges the device operation mode to the game processing mode.Therefore, the player does not need to switch the mode, which alleviatesthe trouble of performing operations before starting the game.

According to the fourteenth aspect, the infrared control means controlsthe infrared radiation device to output an infrared signal in accordancewith an operation on the input device to control a control targetdevice. Therefore, the player can operation another device using theinput device.

According to the fifteenth aspect, the game processing is executed inaccordance with the position information on the infrared light obtainedby the input device. In addition, the infrared light is output from theinfrared radiation device in accordance with the state of the gameprocessing. Therefore, the player can control another device during thegame operation performed using the input device.

These and other features, aspects and advantages of the presenttechnology will become more apparent from the following detaileddescription of the present technology when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view of a game system including a game apparatus 3according to one embodiment;

FIG. 2 is a functional block diagram of the game apparatus 3;

FIG. 3 is an isometric view of a controller 5 seen from the top rearside thereof;

FIG. 4 is an isometric view of the controller 5 seen from the bottomfront side thereof;

FIG. 5A shows an internal structure of the controller 5;

FIG. 5B shows an internal structure of the controller 5;

FIG. 6 is a block diagram showing a structure of the controller 5;

FIG. 7 shows how the controller 5 is used to perform a game operation;

FIG. 8 is an external isometric view of a sensor bar 6 shown in FIG. 1;

FIG. 9 is an isometric view of the sensor bar 6 in the state wherecovers 6RC and 6LC of markers 6R and 6L are removed;

FIG. 10 is a functional block diagram of a control target device;

FIG. 11 shows main data stored on storage means of the game apparatus 3;

FIG. 12 shows an example of signal table data 631;

FIG. 13 shows an example of controllable device data 632;

FIG. 14 shows an example of operation table data 634;

FIG. 15 shows another example of the operation table data 634;

FIG. 16 is a main flowchart illustrating a flow of processing executedby the game apparatus 3;

FIG. 17 is a flowchart illustrating initialization processing (step S1)shown in FIG. 16 in detail;

FIG. 18A shows an exemplary pattern of lighting up LEDs 34 a through 34d;

FIG. 18B shows another exemplary pattern of lighting up the LEDs 34 athrough 34 d;

FIG. 18C shows still another exemplary pattern of lighting up the LEDs34 a through 34 d;

FIG. 19 shows how a TV 2 is controlled by the game apparatus 3 executingthe processing shown in FIG. 16;

FIG. 20 is a flowchart illustrating game processing (step S9) shown inFIG. 16 in detail;

FIG. 21 is a flowchart illustrating course creation processing (stepS31) shown in FIG. 20 in detail;

FIG. 22 shows an exemplary game image displayed on a screen in thecourse creation processing;

FIG. 23 shows how a remote control car 71 is operated using thecontroller 5;

FIG. 24 is a flowchart illustrating race game processing (step S32)shown in FIG. 20 in detail;

FIG. 25 is a flowchart illustrating another game processing (step S9)shown in FIG. 16 in detail;

FIG. 26 shows an exemplary command selection screen image displayed ingame processing in another embodiment of the present technology;

FIG. 27 is a flowchart illustrating a flow of the game processing in theembodiment shown in FIG. 26; and

FIG. 28 shows a modified embodiment of the present technology.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

(Overall Structure of the System)

A game system 1 including a game apparatus according to one embodimentwill be described with reference to FIG. 1. FIG. 1 is an external viewof the game system 1. Hereinafter, a game apparatus and a game programaccording to the present technology will be described using a gameapparatus 3 of an installation type as an example. As shown in FIG. 1,the game system 1 includes a television receiver (hereinafter, referredto simply as the “TV”) 2, the game apparatus 3, an optical disc 4, acontroller 5, and a sensor bar 6. With the game system 1, gameprocessing is performed by the game apparatus 3 based on a gameoperation using the controller 5, while a control target device (forexample, the TV 2) is controlled in accordance with an operation usingthe controller 5. With the game system 1, the controller 5 can be usedfor the game operation and also as a remote control for a control targetdevice which is different from the game apparatus 3.

To the game apparatus 3, the optical disc 4 as an example of anexchangeable information storage medium is detachably inserted. Theoptical disc 4 has stored thereon a game program to be executed by thegame apparatus 3. The game apparatus 3 has an insertion opening for theoptical disc 4 in a front surface thereof. The game apparatus 3 executesgame processing by reading and executing the game program stored on theoptical disc 4 inserted through the insertion opening.

The TV 2 is connected to the game apparatus 3 via a connection cord. TheTV 2 is a display device such as a home-use television receiver or thelike. The TV 2 displays a game image obtained as a result of the gameprocessing executed by the game apparatus 3. The sensor bar 6 isprovided in the vicinity of the TV 2 (in FIG. 1, above a screen of theTV 2). The sensor bar 6 includes two markers 6R and 6L at both endsthereof. Specifically, the markers 6R and 6L each include one or moreinfrared LEDs, and output infrared light forward from the TV 2. Thesensor bar 6 is connected to the game apparatus 3, and the gameapparatus 3 is capable of controlling each of the infrared LEDs includedin the sensor bar 6 to be on or off. In this embodiment, the markers 6Rand 6L are used for the game operation using the controller 5 and alsoused as means for transmitting an infrared signal for controlling thecontrol target device (for example, the TV 2).

The controller 5 is an input device for providing the game apparatus 3with operation data which represents operation particulars made on thecontroller 5. The controller 5 and the game apparatus 3 wirelesslycommunicate with each other. In this embodiment, the Bluetooth(registered trademark) technology is used for such wirelesscommunication. The controller 5 transmits the operation data to the gameapparatus 3, and the game apparatus 3 executes the game processing inaccordance with the received operation data. The controller 5 alsoincludes imaging means (imaging information calculation section 35 shownin FIG. 6) for taking an image of the markers 6R and 6L and accelerationdetection means (acceleration sensor 37), in addition to operationbuttons. As described later in detail, the game apparatus 3 receivesdata obtained by the imaging means and the acceleration detection meansas operation data, and thus can calculate a position, posture or motionof the controller 5. Owing to this, the player can perform a gameoperation of moving the controller 5 itself as well as an operation madeon the control buttons. The game apparatus 3 transmits control data forcontrolling the motion of the controller 5. In accordance with thereceived control data, the controller 5 performs a motion of, forexample, outputting a sound from a speaker provided in the controller 5or lighting up an LED provided in the controller 5. In anotherembodiment, the controller 5 and the game apparatus 3 may be connectedto each other in a wired manner.

(Structure of the Game Apparatus)

Next, with reference to FIG. 2, a structure of the game apparatus 3 willbe described. FIG. 2 is a functional block diagram of the game apparatus3.

As shown in FIG. 2, the game apparatus 3 includes, for example, a CPU(central processing unit) 10 for executing various programs. The CPU 10executes a start program stored on a boot ROM (not shown) to, forexample, initialize memories including a main memory 13, and thenexecutes a game program stored on the optical disc 4 to perform gameprocessing or the like in accordance with the game program. The CPU 10is connected to a GPU (Graphics Processing Unit) 12, the main memory 13,a DSP (Digital Signal Processor) 14, and an ARAM (Audio RAM) 15 via amemory controller 11. The memory controller 11 is connected to acommunication unit 16, a video I/F (interface) 17, a flash memory 18, anLED control section 19, an audio I/F 20 and a disc I/F 21 via apredetermined bus. The video I/F 17 is connected to the TV 2, the LEDcontrol section 19 is connected to the sensor bar 6, the audio I/F 20 isconnected to a speaker 55 and a disc drive 22, and the disc I/F 21 isconnected to the disc drive 22.

The GPU 12 performs image processing based on an instruction from theCPU 10. The GPU 12 includes, for example, a semiconductor chip forperforming calculation processing necessary for displaying 3D graphics.The GPU 12 performs the image processing using a memory dedicated forimage processing (not shown) and a part of the storage area of the mainmemory 13. The GPU 12 generates game image data and a movie to bedisplayed on the TV 2 using such memories, and outputs the generateddata or movie to the TV 2 via the memory controller 11 and the video I/F17 as necessary.

The main memory 13 is a storage area used by the CPU 10, and stores agame program or the like necessary for processing performed by the CPU10 as necessary. For example, the main memory 13 stores a game programread from the optical disc 4 by the CPU 10, various types of data or thelike. The game program, the various types of data or the like stored onthe main memory 13 are executed by the CPU 10.

The DSP 14 processes sound data or the like generated by the CPU 10during the execution of the game program. The DSP 14 is connected to theARAM 15 for storing the sound data or the like. The ARAM 15 is used whenthe DSP 14 performs predetermined processing (for example, storage ofthe game program or sound data already read). The DSP 14 reads the sounddata stored on the ARAM 15 and outputs the sound data to the speaker 55via the memory controller 11 and the audio I/F 20.

The memory controller 11 comprehensively controls data transfer, and isconnected to the communication unit 16, the flash memory 18, the LEDcontrol section 19, and the various I/Fs 17, 20 and 21. Thecommunication unit 16 receives operation data from the controller 5, andoutputs the received operation data to the CPU 10 via the memorycontroller 11. When control data for controlling the motion of thecontroller 5 is transmitted to the controller 5, the control data isoutput to the communication unit 16. The communication unit 16 transmitsthe input control data to the controller 5. The video I/F 17 isconnected to the TV 2. The image data generated by the GPU 12 is outputto the TV 2 via the video I/F 17. The flash memory 18 acts as a backupmemory for fixedly storing data such as saved data or the like. The gameapparatus 3 can reproduce a game state in the past and display the gameimage on the TV 2 using the saved data stored on the flash memory 18.The LED control section 19 is connected to the infrared LEDs included inthe sensor bar 6. For lighting up the infrared LEDs, the CPU 10instructs the LED control section 19 to supply power. In accordance withthe instruction, the LED control section 19 supplies power to theinfrared LEDs, and thus the infrared LEDs are lit up. The audio I/F 20is connected to the speaker 55 built in the TV 2. The sound data read bythe DSP 14 from the ARAM 15 or sound data directly output from the discdrive 22 is output from the speaker 55. The disc I/F 21 is connected tothe disc drive 22. The disc drive 22 reads data stored at apredetermined reading position of the optical disc 4, and outputs thedata to the disc I/F 21 and the audio I/F 20.

The game apparatus 3 includes a network communication section (notshown) and thus is connected to a network such as the Internet or thelike. The game apparatus 3 can obtain various data from outside ortransmit data to the outside via the network communication section.

(Structure of the Controller)

With reference to FIG. 3 through FIG. 7, the controller 5 will bedescribed. FIG. 3 and FIG. 4 are external isometric views of thecontroller 5. FIG. 3 is an isometric view of the controller 5 seen fromthe top rear side thereof. FIG. 4 is an isometric view of the controller5 seen from the bottom front side thereof.

As shown in FIG. 3 and FIG. 4, the controller 5 includes a housing 31formed by plastic molding or the like. The housing 31 has a generallyparallelepiped shape extending in a longitudinal or front-rear direction(Z-axis direction shown in FIG. 3). The overall size of the housing 31is small enough to be held by one hand of an adult or even a child. Theplayer can perform a game operation by, for example, pressing buttonsprovided in the controller 5 or moving the controller 5 itself to changethe position or posture thereof. For example, the player can perform anoperation on an operation target by rotating the controller 5 around anaxis in the longitudinal direction thereof or changing the positionindicated by the controller 5 on the display screen. The “positionindicated by the controller 5 on the display screen” ideally refers to aposition at which a phantom straight line extending from a front end ofthe controller 5 in the longitudinal direction crosses the displayscreen of the TV 2. However, the “position indicated by the controller 5on the display screen” does not need to be exactly such a position, butmay be a position in the vicinity thereof which can be calculated by thegame apparatus 3. Hereinafter, such a position will be sometimesreferred to as an “indicated position” or an “indicated position by thecontroller 5”. The longitudinal direction of the controller 5 (housing31) will be sometimes referred to as an “indicated direction” or an“indicated direction by the controller 5”.

The housing 31 has a plurality of operation buttons. As shown in FIG. 3,provided on a top surface of the housing 31 are a cross key 32 a, afirst button 32 b, a second button 32 c, an A button 32 d, a minusbutton 32 e, a home button 32 f, a plus button 32 g, and a power button32 h. The power button 32 h is for remotely turning the game apparatus 3on or off. As shown in FIG. 4, on a bottom surface of the housing 31, arecessed portion is formed. On a rear slope surface of the recessedportion, a B button 32 i is provided. These buttons 32 a through 32 iare assigned various functions in accordance with the game programexecuted by the game apparatus 3. The home button 32 f and the powerbutton 32 h have a surface thereof buried in the top surface of thehousing 31, so as not to be inadvertently pressed by the player.

On a rear surface of the housing 31, a connector 33 is provided. Theconnector 33 is used for connecting the controller 5 to another device.For example, a sub control unit including a stick inclinable at anyangle in the range of 360 degrees may be connected to the connector 33via a cable. In this way, for example, a direction may be input inaccordance with an operation on the sub control unit, while apredetermined position on the screen may be indicated by an operationmade on the controller 5. By using such a sub control unit connected tothe connector 33 via a cable, an operation of inputting a directionwhile moving the controller 5 can be freely performed.

In a rear part of the top surface of the housing 31, a plurality of LEDs(in FIG. 3, four LEDs 34 a through 34 d) are provided. The controller 5is assigned a controller type (number) so as to be distinguishable fromthe other controllers 5. The LEDs 34 are used for, for example,informing the player of the controller type which is currently set tocontroller 5 that he/she is using, or for informing the player of theremaining battery amount. Specifically, when the controller 5 transmitsthe operation data to the game apparatus 3, one of the plurality of LEDs34 a through 34 d corresponding to the controller type is lit up.

The controller 5 includes the imaging information calculation section 35(FIG. 5B). As shown in FIG. 4, a light incident face 35 a of the imaginginformation calculation section 35 is provided on a front surface of thehousing 31. The light incident face 35 a is formed of a material whichallows infrared light from the markers 6R and 6L to be at leasttransmitted therethrough.

On the top surface of the housing 31, sound holes 31 a are formedbetween the first button 32 b and the home button 32 f for releasing thesound outside from a speaker 49 (FIG. 5A) built in the controller 5.

With reference to FIG. 5A and FIG. 5B, an internal structure of thecontroller 5 will be described. FIG. 5A and FIG. 5B illustrate aninternal structure of the controller 5. FIG. 5A is an isometric viewillustrating a state where an upper casing (a part of the housing 31) ofthe controller 5 is removed. FIG. 5B is an isometric view illustrating astate where a lower casing (a part of the housing 31) of the controller5 is removed. FIG. 5B shows a reverse side of a substrate 30 shown inFIG. 5A.

As shown in FIG. 5A, the substrate 30 is fixed inside the housing 31. Ona top main surface of the substrate 30, the operation buttons 32 athrough 32 h, the LEDs 34 a through 34 d, the acceleration sensor 37, anantenna 45, the speaker 49 and the like are provided. These elements areconnected to a microcomputer 42 (see FIG. 5B) via lines (not shown)formed on the substrate 30 and the like. In this embodiment, theacceleration sensor 37 is provided off the center line of the controller5 in the X-axis direction. This makes it easier to calculate the motionof the controller 5 when the controller is rotated around the Z axis asthe rotation center. A wireless module 44 (FIG. 6) and the antenna 45allow the controller 5 to act as a wireless controller.

As shown in FIG. 5B, at a front edge of a bottom main surface of thesubstrate 30, the imaging information calculation section 35 isprovided. The imaging information calculation section 35 includes aninfrared filter 38, a lens 39, an imaging element 40 and an imageprocessing circuit 41 located in this order from the front surface ofthe controller 5. These elements are attached to the bottom main surfaceof the substrate 30.

On the bottom main surface of the substrate 30, the microcomputer 42 anda vibrator 48 are provided. The vibrator 48 may be, for example, avibration motor or a solenoid, and is connected to the microcomputer 42via lines provided on the substrate 30 and the like. The controller 5 isvibrated by an actuation of the vibrator 48 based on an instruction fromthe microcomputer 42, and the vibration is conveyed to the playerholding the controller 5. Thus, a so-called vibration-responsive game isrealized. In this embodiment, the vibrator 48 is provided in a frontpart of the housing 31. Since the vibrator 48 is provided closer to afront end than the center of the controller 5, the vibration of thevibrator 48 can vibrate the entire controller 5 more significantly. Theconnector 33 is attached at a rear edge of the main bottom surface ofthe substrate 30. In addition to the elements shown in FIG. 5A and FIG.5B, the controller 5 includes a quartz oscillator for generating areference clock of the microcomputer 42, an amplifier for outputting anaudio signal to the speaker 49, and the like.

The shape of the controller 5, the shape of the operation buttons, andthe number, position or the like of the acceleration sensor and thevibrator shown in FIG. 3 through FIG. 5B are merely exemplary, and maybe altered without departing from the scope of the present technology.The position of the imaging information calculation section 35 (thelight incident face 35 a of the imaging information calculation section35) in the controller 5 does not need to be on the front surface of thehousing 31, and may be on another surface as long as light can enterfrom the outside of the housing 31. In this case, the “indicateddirection by the controller 5” is the imaging direction of the imagingelement 40, more specifically a direction vertical to the light incidentface.

FIG. 6 is a block diagram showing a structure of the controller 5. Thecontroller 5 includes the operation section 32 (operation buttons), theimaging information calculation section 35, the communication section36, the acceleration sensor 37, the speaker 49, and the LEDs 34 athrough 34 d.

The operation section 32 corresponds to the above-described operationbuttons 32 a through 32 i, and outputs data representing an input stateof each of the operation buttons 32 a through 32 i (whether each of theoperation buttons 32 a through 32 i has been pressed or not) to themicrocomputer 42 of the communication section 36.

The imaging information calculation section 35 is a system for analyzingimage data taken by imaging means, distinguishing an area having a highbrightness in the image data, and calculating the center of gravity, thesize and the like of the area. The imaging information calculationsection 35 has, for example, a maximum sampling period of about 200frames/sec., and therefore can trace and analyze even a relatively fastmotion of the controller 5.

The imaging information calculation section 35 includes the infraredfilter 38, the lens 39, the imaging element 40 and the image processingcircuit 41. The infrared filter 38 allows only infrared light to passtherethrough, among light incident on the front surface of thecontroller 5. The lens 39 collects the infrared light which has beentransmitted through the infrared filter 38 and causes the infrared lightto be incident on the imaging element 40. The imaging element 40 is asolid-state imaging device such as, for example, a CMOS sensor or a CCDsensor. The imaging element 40 receives the infrared light collected bythe lens 39 and outputs an image signal. The markers 6R and 6L of thesensor bar 6 located in the vicinity of the display screen of the TV 2each include an infrared LED for outputting infrared light forward fromthe TV 2. The provision of the infrared filter 38 allows the imagingelement 40 to receive only the infrared light transmitted through theinfrared filter 38 to generate image data. Therefore, the image of eachof the markers 6R and 6L can be taken more accurately. Hereinafter, animage taken by the imaging element 40 will be referred to as a “takenimage”. The image data generated by the imaging element 40 is processedby the image processing circuit 41. The image processing circuit 41calculates the positions of imaging targets (the markers 6R and 6L) inthe taken image. Hereinafter, a method for calculating the positions ofthe imaging targets will be described.

When the taken image is input from the imaging element 40 to the imageprocessing circuit 41, the image processing circuit 41 calculates acoordinate set representing the position of each of areas in the takenimage which match a predetermined condition. Here, the predeterminedcondition is a condition for specifying an image of an imaging target(target image). A specific predetermined condition is that the area hasa brightness of a predetermined value or greater (a high luminance area)and has a size within a predetermined size range. The predeterminedcondition only needs to be a condition for specifying an imaging target,and in another embodiment, may include a condition regarding the colorof the image.

For calculating the position of the target image, the image processingcircuit 41 specifies high brightness areas described above, from theareas in the taken image, as candidates for the target image. The reasonis that a target image appears as a high brightness area in the imagedata of the taken image. Next, based on the size of each specified highbrightness area, the image processing circuit 41 executes determinationprocessing of determining whether or not each of the high brightnessareas is a target image. The taken image may include images other thanimages of the markers 6R and 6L as the target images, due to sunlightcoming through a window or light of a fluorescent lamp. In this case,the images other than the images of the markers 6R and 6L also appear ashigh brightness areas. The above-mentioned determination processing isexecuted in order to distinguish the images of the markers 6R and 6L asthe target images from the other images, so that the target images areaccurately specified. Specifically, it is determined whether or not eachspecified high brightness area has a size within a predetermined sizerange. When the high brightness area has a size within the predeterminedsize range, such an area is determined to represent a target image;whereas when the high brightness area has a size outside thepredetermined size range, such an area is determined to represent animage other than a target image.

The image processing circuit 41 calculates the position of a highbrightness area which is determined to represent a target image as aresult of the determination. Specifically, the image processing circuit41 calculates the position of the center of gravity of the highbrightness area. The position of the center of gravity can be calculatedin a scale more detailed than the resolution of the imaging element 40.For example, even when the resolution of a taken image taken by theimaging element 40 is 126×96, the position of the center of gravity iscalculated at a scale of 1024×768. The coordinate set of the position ofthe center of gravity is represented by integers of (0, 0) to (1024,768). Positions in the taken image are represented by a coordinatesystem (X-Y coordinate system), in which the upper left corner of thetaken image is the origin, the downward direction from the origin is apositive Y-axis direction, and the rightward direction from the originis a positive X-axis direction.

As described above, the image processing circuit 41 calculates thecoordinate set representing the position of each of areas in the takenimage which match the predetermined condition. The image processingcircuit 41 outputs the calculated coordinate set to the microcomputer 42of the communication section 36. Data on the coordinate set istransmitted to the game apparatus 3 as operation data by themicrocomputer 42. Hereinafter, such a coordinate set will be referred toas a “marker coordinate set”. Since the marker coordinate set varies inaccordance with the direction (posture) or position of the controller 5itself, the game apparatus 3 can calculate the direction or position ofthe controller 5 using the marker coordinate set.

Returning to FIG. 6, the acceleration sensor 37 detects an acceleration(including acceleration of gravity) of the controller 5. Namely, theacceleration sensor 37 detects a force (including the force of gravity)applied to the controller 5. The acceleration sensor 37 detects a valueof the acceleration in a linear direction along a sensing axis among theaccelerations acting on a detection section of the acceleration sensor37. For example, in the case of a multi-axial (at least two-axial)acceleration sensor, an acceleration of a component along each axis(linear acceleration) is detected as an acceleration acting on thedetection section of the acceleration sensor. For example, a three-axialor two-axial acceleration sensor 37 may be available from AnalogDevices, Inc. or STMicroelectronics N.V.

In this embodiment, the acceleration sensor 37 detects a linearacceleration in each of an up-down direction with respect to thecontroller 5 (Y-axis direction shown in FIG. 3), a left-right directionwith respect to the controller 5 (X-axis direction shown in FIG. 3), anda front-rear direction with respect to the controller 5 (Z-axisdirection shown in FIG. 3). Since the acceleration sensor 37 detects anacceleration in the linear direction along each axis, the output fromthe acceleration sensor 37 represents a value of the linear accelerationalong each of the three axes. Namely, the detected acceleration isrepresented as a three-dimensional vector in an x-y-z coordinate systemwhich is set with respect to the controller 5. Data representing theacceleration detected by the acceleration sensor 37 (acceleration data)is output to the communication section 36. In this embodiment, thecommunication section 36 of the controller 5 outputs acceleration datato the game apparatus 3 at a constant interval (for example, every 0.5ms). The game apparatus 3 can calculate the moving direction or theinclination (posture) of the controller 5 based on the accelerationdata. Since the acceleration sensor 37 detects a linear component ofacceleration along each of the axes, the game apparatus 3 cannotdirectly detect the moving direction or the inclination of thecontroller 5. The moving direction or the inclination of a device havingthe acceleration sensor 37 mounted thereon is calculated by executingpredetermined calculation processing on the acceleration detected foreach axis of the acceleration sensor 37.

The communication section 36 includes the microcomputer 42, a memory 43,the wireless module 44 and the antenna 45. The microcomputer 42 controlsthe wireless module 44 for wirelessly transmitting the data obtained bythe microcomputer 42 while using the memory 43 as a storage area duringprocessing.

Data which is output from the operation section 32, the imaginginformation calculation section 35, and the acceleration sensor 37 tothe microcomputer 42 is temporarily stored on the memory 43. Thewireless transmission from the communication section 36 to thecommunication unit 16 is performed at a predetermined time interval.Since game processing is generally performed at a cycle of 1/60 sec. (ata cycle of one frame), the wireless transmission is preferably performedat a cycle of a time period equal to or shorter than 1/60 sec. At thetransmission timing to the communication unit 16, the microcomputer 42outputs the data stored on the memory 43 to the wireless module 44 asoperation data. The wireless module 44 uses, for example, the Bluetooth(registered trademark) technology to modulate a carrier wave of apredetermined frequency with the operation data and radiate theresultant very weak electric signal from the antenna 45. Namely, theoperation data is modulated into a very weak electric signal by thewireless module 44 and transmitted from the controller 5. The very weakelectric signal is received by the communication unit 16 on the side ofthe game apparatus 3. The received very weak electric signal isdemodulated or decoded, so that the game apparatus 3 can obtain theoperation data. The CPU 10 of the game apparatus 3 executes the gameprocessing based on the obtained operation data and the game program.

The game apparatus 3 transmits sound data to the controller 5 at anappropriate timing in accordance with the situation in the game. Thesound data transmitted from the game apparatus 3 is received by theantenna 45. The microcomputer 42 obtains the sound data received by theantenna 45 via the wireless module 44. The microcomputer 42 alsoperforms predetermined processing on the obtained sound data and outputsan audio signal to the speaker 49 via an amplifier (not shown). Thus,the game apparatus 3 can output a sound such as a sound effect of thegame or the like from the speaker 49 on the side of the controller 5.

The microcomputer 42 controls the LEDs 34 a through 34 d to be on oroff. For example, the microcomputer 42 detects the remaining batteryamount of the controller 5, and lights up a part of, or all of, the LEDs34 a through 34 d based on the detection result. Accordingly, the playercan visually check the remaining battery amount with the LEDs 34 athrough 34 d. The microcomputer 42 may light up a part of, or all of,the LEDs 34 a through 34 d in accordance with an instruction from thegame apparatus 3 (the above-described control data).

By using the controller 5, the player can perform a game operation ofchanging the posture of the controller 5, moving the position of thecontroller 5 or rotating the controller 5, in addition to a conventionalgeneral operation of pressing the operation buttons.

FIG. 7 shows how the controller 5 is used to perform a game operation.When playing the game using the controller 5 with the game system 1, theplayer holds the controller 5 with one hand as shown in FIG. 7. In thisembodiment, the player performs a game operation by designating anyposition on the display screen of the TV 2 with the controller 5(designating a desired position on the display screen as the indicatedposition by the controller 5).

(Structure of the Sensor Bar)

With reference to FIG. 8 and FIG. 9, the sensor bar 6 will be described.FIG. 8 is an external isometric view of the sensor bar 6 shown inFIG. 1. As shown in FIG. 8, the sensor bar 6 has a rod-like externalshape. The sensor bar 6 includes two markers, i.e., the marker 6R on oneof two ends thereof and the marker 6L on the other end thereof. Themarker 6R includes a cover 6RC and an infrared LED (an infrared LED 6Rashown in FIG. 9). The marker 6L includes a cover 6LC and an infrared LED(an infrared LED 6La shown in FIG. 9).

FIG. 9 is an isometric view of the sensor bar 6 in the state where thecover 6RC of the marker 6R and the cover 6LC of the marker 6L areremoved. As shown in FIG. 9, the marker 6R includes four infrared LED6Ra, and the marker 6L includes four infrared LED 6La. The four infraredLED 6Ra are located close together, and so are imaged integrally by theimaging means of the controller 5. The four infrared LED 6Ra are locatedin one line laterally and are directed radially. Namely, the two outerinfrared LEDs 6Ra are directed outward than the two inner infrared LEDs6Ra. Owing to this arrangement, the radiation angle of the entirety ofthe four infrared LEDs 6Ra (the radiation angle of the marker 6R) isincreased in the lateral direction. Therefore, where the sensor bar 6 isinstalled in the vicinity of the TV 2, the controller 5 can receive theinfrared light from the marker 6R in a wide right-to-left range withrespect to the direction perpendicular to the display screen of the TV2. Namely, the player can use the controller 5 in the wide right-to-leftrange with respect to the direction perpendicular to the display screenof the TV 2. The four infrared LEDs 6La are located in the same manneras the infrared LEDs 6Ra. In another embodiment, each marker may includeany number of infrared LEDs. In the case where each marker includes aplurality of infrared LEDs, the infrared LEDs are preferably located inone line laterally such that the LEDs are directed radially. Each markermay include one infrared LED and a cover for randomly reflecting theinfrared light from the infrared LED.

(Structure of the Control Target Device)

Next, with reference to FIG. 10, the TV 2 as an example of the controltarget device will be described. FIG. 10 is a functional block diagramof the TV 2. As shown in FIG. 10, the TV 2 includes an operation section51, an infrared receiving section 52, a microcomputer 53, a displaysection 54, and a speaker 55. The operation section 51 is input meansfor operating a power on/off switch, a channel switch, a volume switchand other functional elements of the TV 2. An operation signal whichrepresents an operation made on each switch of the operation section 51is output to the microcomputer 53. The infrared receiving section 52receives an infrared signal (remote control signal) from a remotecontrol of the TV 2 or the sensor bar 6. The received infrared signal isconverted into an electric signal and output to the microcomputer 53.The microcomputer 53 controls the display section 54, the speaker 55 andother elements not shown including a tuner, in accordance with thesignal from the operation signal from the operation section 51 or asignal from the infrared receiving section 52. For example, whenreceiving a signal to switch the power on/off, the microcomputer 53stops power supply to the corresponding element. When receiving a signalto change the sound volume, the microcomputer 53 changes the volume ofthe sound which is output from the speaker 55. FIG. 10 shows the TV 2 asan example of the control target device, but the control target devicemay be any device which includes an infrared receiving section and themotion of which is controllable by an infrared signal.

(Processing by the Game Apparatus)

Next, processing executed by the game apparatus 3 will be described. Inthis embodiment, a race game is executed by the game apparatus 3. Inthis embodiment, the TV 2 and a remote control car 71 (described laterwith reference to FIG. 23) are controlled as the control target devicesby the game apparatus 3.

First, main data used by the game apparatus 3 for the game processingwill be described with reference to FIG. 11. FIG. 11 shows main datastored on storage means (the main memory 13, the flash memory 18, andthe like) of the game apparatus 3. As shown in FIG. 11, the storagemeans of the game apparatus 3 has stored thereon a game program 61,operation data 62, processing data 63 and the like. The main memory 13includes image data on characters appearing in the game, sound data onsound effects and BGM, and other data necessary for the game processingin addition to the data shown in FIG. 11.

The game program 61 is partially or entirely read from the optical disc4 and stored on the main memory 13 at an appropriate timing after theoptical disc 4 is inserted into the game apparatus 3. The game program61 includes programs necessary for executing the game processingdescribed later.

The operation data 62 is transmitted from the controller 5 to the gameapparatus 3, and is stored on the main memory 13. The operation data 62includes marker coordinate set data 621, operation button data 622, andacceleration data 623. The marker coordinate set data 621 represents thepositions of the imaging targets (the markers 6R and 6L) in the takenimage, i.e., the marker coordinate sets mentioned above. The operationbutton data 622 represents operation particulars performed on each ofthe buttons 32 a through 32 i of the operation section 32 (whether ornot each of the buttons 32 a through 32 i has been pressed). Theacceleration data 623 represents an output from the acceleration sensor37. In this embodiment, the acceleration data 623 is not used for thegame processing, and so the controller 5 may have a structure withoutthe acceleration sensor 37.

The processing data 63 is used for the processing executed by the gameapparatus 3 (see FIG. 16 and the like). The processing data 63 includessignal table data 631, controllable device data 632, control target data633, operation table data 634, input history data 635 and race coursedata 636.

The signal table data 631 represents correspondence information betweenan instruction on the control target device and a signal pattern of theinfrared signal to be transmitted in order to cause the control targetdevice to follow the instruction. FIG. 12 shows an example of the signaltable data 631. As shown in FIG. 12, the signal table data 631 includesthe correspondence information between the instruction and the signalpattern for each device. The signal table data 631 includes differentcorrespondence information for different types (e.g., manufacturer ormodel number) of the same device (e.g., TV). In this embodiment, the TV2 and the remote control car 71 are the control target devices. Thus,the signal table data 631 includes correspondence information regardingthe TV 2 and the remote control car 71.

In FIG. 12, device A is the TV 2, which is one of the control targetdevices. For device A, an instruction for switching power on/off andsignal pattern A are associated with each other. In addition, aninstruction for increasing the sound volume by one stage and signalpattern B are associated with each other.

In FIG. 12, device B is the remote control car 71, which is one of thecontrol target devices. Herein, the game apparatus 3 can control anacceleration amount and the angle of the steering wheel of the remotecontrol car 71. Specifically, as shown in FIG. 12, for device B, aninstruction for turning on the accelerator and signal pattern C areassociated with each other. In addition, an instruction for setting theangle of the steering wheel to θ1 and signal pattern E are associatedwith each other, and an instruction for setting the angle of thesteering wheel to θ2 and signal pattern F are associated with eachother. The angle of the steering wheel of the device B (remote controlcar 71) is controllable at, for example, 9 stages of θ1 through θ9, anda signal pattern is associated with each angle of the steering wheel.

The signal table data 631 is prepared in advance and is stored on, forexample, the flash memory 18. The game apparatus 3 may update thecontents of the signal table data 631 by obtaining correspondenceinformation on a new device from a network or the like. In this case,even when a new control target device is added as a control targetdevice, the correspondence information on such a device can be easilyobtained.

FIG. 13 shows an example of the controllable device data 632. Thecontrollable device data 632 represents information on whether each ofthe devices described in the signal table data 631 is usable or not as acontrol target device (usability information). The usability informationis set (registered) by the player in advance. More specifically, theusability information is initially set to “non-usable”. The term“non-usable” means that the device is not usable as a control targetdevice. The player selects devices to be used as control target devicesfrom a plurality of devices described in the controllable device data632. The game apparatus 3 sets the usability information of the selecteddevices to “usable”. During the processing shown in FIG. 16, the gameapparatus 3 selects a control target device from the devices which areset to “usable”. In this embodiment, as shown in FIG. 13, device A anddevice B are set to “usable”, and the other devices (for example, deviceC) are set to “non-usable”. The game apparatus 3 may set the usabilityinformation of only one type of a certain device (e.g., TV) to “usable”and set the other types of the same device to “non-usable”.

The control target data 633 represents a device which is currently setas the control target device. Specifically, the control target data 633represents one of the devices which are set to “usable” in thecontrollable device data 632. When none of the devices are “usable”, thecontrol target data 633 indicates that there is no control targetdevice. In this embodiment, the control target device may be set orchanged automatically by the game apparatus 3 at a predetermined timing.In another embodiment, the player may set or change the control targetdevice.

The operation table data 634 represents correspondence between anoperation using the controller 5 and an instruction to be followed bythe control target device when such an operation is performed. FIG. 14and FIG. 15 show examples of the operation table data 634. FIG. 14 showsthe operation table data 634 when the control target device is, forexample, device A (TV) shown in FIG. 12. In FIG. 14, an instruction forswitching the power on/off is associated with an operation of pressingthe A button 32 d of the controller 5. In addition, an instruction forincreasing the sound volume by one stage is associated with an operationof pressing the plus button 32 g of the controller 5. When the controltarget device is changed during the processing by the game apparatus 3,the contents of the operation table data 634 are changed accordingly.

FIG. 15 shows the operation table data 634 when the control targetdevice is, for example, device B (remote control car) shown in FIG. 12.In FIG. 15, an instruction for turning the accelerator on is associatedwith an operation of pressing the A button 32 d of the controller 5. Inaddition, an instruction for setting the angle of the steering wheel toθ1 is associated with an operation of limiting the inclination angle ofthe controller 5 when the controller 5 is rotated around an axis in thelongitudinal direction thereof to a range of θa to θb. Herein, theinclination angle of the controller 5 (0° through) 360° and the angle ofthe steering wheel are associated with each other such that the angle ofthe steering wheel is uniquely determined in accordance with theinclination angle of the controller 5. The inclination angle of thecontroller 5 when the controller 5 is rotated around an axis in thelongitudinal direction thereof can be calculated based on the angle ofthe line segment connecting the marker coordinate sets of the twomarkers 6R and 6L. The inclination angle of the controller 5 isrepresented with respect to a predetermined reference angle of 0°.

As shown in FIG. 14 and FIG. 15, the contents of the operation tabledata 634 are different depending on the control target device. When thecontrol target device is changed during the processing by the gameapparatus 3, the contents of the operation table data 634 are changedaccordingly.

The input history data 635 shows the history of inputs to the controller5 used for operating the characters appearing in the game space or thecontrol target device. The input history data 635 may be anything whichspecifies the inputs performed on the controller 5. For example, theinput history data 635 may be a part or the entirety of the operationdata 62 transmitted from the controller 5. When representing inputs usedfor operating the control target device, the input history data 635 mayrepresent an instruction given to the control target device. Whenrepresenting inputs used for operating a character, the input historydata 635 may represent a motion of the character (for example, theposition, speed, etc. of the character).

The race course data 636 represents a race course constructed in avirtual game space. Before the race game is started, the game apparatus3 constructs a race course in the virtual game space in accordance withthe race course data 636. A racing car operated by the player runs onthe constructed race course.

Next, the processing executed by the game apparatus 3 will be describedwith reference to FIG. 16 through FIG. 25 in detail. FIG. 16 is a mainflowchart illustrating a flow of the processing executed by the gameapparatus 3. In this embodiment, the game apparatus 3 is in a stand-bystate before being turned on. In this stand-by state, the game apparatus3 stops executing the game program stored on the optical disc 4, and iscapable of executing processing of, for example, obtaining various datafrom the network. The game apparatus 3 is capable of receiving data fromthe controller 5 even in the stand-by state. In the case where the gameapparatus 3 and the controller 5 communicate each other wirelessly usingthe Bluetooth (registered trademark) technology, it is necessary toexecute so-called paring processing in order to allow the game apparatus3 and the controller 5 to recognize each other. In this embodiment, itis assumed that the paring processing is already executed and thewireless communication is possible between the game apparatus 3 and thecontroller 5.

When the power button 32 h of the controller 5 is pressed while the gameapparatus 3 is in the stand-by state, the game apparatus 3 is turned on.Namely, the game apparatus 3, upon receiving operation data 62 whichrepresents that the power button 32 h has been pressed from thecontroller 5, terminates the stand-by state and is turned on. When thegame apparatus 3 is turned on, the CPU 10 of the game apparatus 3executes the start program stored on the boot ROM (not shown) toinitialize the elements including the main memory 13. The processingshown in FIG. 16 is executed after such processing is completed. In thisembodiment, the programs for executing processing other than the gameprocessing (step S9) among the processing shown in FIG. 16 are stored inthe game apparatus 3 in advance. In another embodiment, such programsmay be stored on the optical disc 4.

In this embodiment, before the processing shown in FIG. 16 is executed,a device having a TV function is set as the control target device in thecontrol target data 633 among the devices usable as the control targetdevices. The devices usable as the control target devices arerepresented by the controllable device data 632. In this embodiment, thedevices to be used as the control target devices are the TV 2 and theremote control car 71. Data representing the TV 2 is stored on the mainmemory 13 as the control target data 633. The contents of the operationtable data 634 are updated such that a function for operating the TV 2is assigned to at least one of the operation buttons 32 a through 32 iof the controller 5. When there is no device having a TV function amongthe devices usable as the control target devices, data indicating thatthere is no control target device is stored on the main memory 13 as thecontrol target data 633.

With reference to FIG. 16, initialization processing is executed in stepS1. The initialization processing is executed for identifying thecontroller 5 used in the subsequent processing. Hereinafter, withreference to FIG. 17, the initialization processing will be described.

FIG. 17 is a flowchart illustrating the initialization processing (step1) shown in FIG. 16 in detail. The initialization processing is executedas follows. In step S21, the CPU 10 starts connection processing withthe controller 5. By the connection processing, the game apparatus 3assigns a number to the controller 5 for identifying the controller 5.The CPU 10 detects the controllers 5 which are currently wirelesslycommunicable with the game apparatus 3, and assigns an inherent numberto each of the detected controllers 5 (for example, sequentially assignsinteger numbers from “1”). In this embodiment, the maximum number ofcontrollers 5 which are wirelessly communicable at once is four.

Next in step S22, the CPU 10 determines whether or not the connectionprocessing has been completed. When the determination result in step S22is positive, processing in step S23 is executed. By contrast, when thedetermination result in step S22 is negative, processing in step S22 isrepeated. Namely, the CPU 10 waits until the connection processing iscompleted, and executes the processing in step S23 after the connectionprocessing is completed.

In this embodiment, while the connection processing is being executed,each of the controllers 5 notifies the remaining battery amount thereofto the player. Specifically, the microcomputer 42 of each controller 5detects the remaining battery amount and outputs informationrepresenting the detection result using the LEDs 34 a through 34 d. Inmore detail, the number of LEDs to be lit up is changed in accordancewith the detected remaining battery amount. For example, as theremaining battery amount is larger, a greater number of LEDs are lit up.Owing to this operation, the player can confirm the remaining batteryamount of his/her controller 5 while waiting for the controller 5 to beconnected to the game apparatus 3. In the connection processing, uponreceiving the number assigned to each controller 5, the microcomputer 42of the controller 5 stops lighting up the LEDs 34 a through 34 d torepresent the remaining battery amount. Then, the microcomputer 42outputs information representing the assigned number using the LEDs 34 athrough 34 d. Specifically, the microcomputer 42 lights up only the LEDcorresponding to the assigned number among the LEDs 34 a through 34 d.In the case where a plurality of controllers 5 are assigned a number bythe game apparatus 3, each controller 5 is given a different number.Thus, the plurality of controllers 5 have different LEDs lit up from oneanother. Owing to this, the player can easily identify his/hercontroller from the controllers 5 of the other players.

In step S23, the CPU 10 generates image data for an initial screenimage. The image data for the initial screen image may be stored inadvance on the storage means such as the flash memory 18 or the like.The initial screen image is a menu screen image for, for example,displaying executable game programs or controllable control targetdevices, or for allowing the player to perform various settingsregarding the game apparatus 3. When the TV 2 is on at the time of stepS23, the initial screen image is displayed on the TV 2. When the TV 2 isoff at the time of step S23, the initial screen image is not displayedon the TV 2. In this case, the initial screen image is displayed on theTV 2 after the TV is turned on. After step S23, the CPU 10 terminate theinitialization processing.

Returning to FIG. 16, in step S2, the CPU 10 determines whether or not aremote control function of the game system 1 is valid. Herein, theexpression that “the remote control function is valid” means that thereis at least one controllable control target device. When the controltarget data 633 stored on the main memory 13 represents a device, it isdetermined that the remote control function is valid. By contrast, whenthe control target data 633 stored on the main memory 13 does notrepresent any device (i.e., when the control target data 633 indicatesthat there is no control target device), it is determined that theremote control function is invalid. When the determination result instep S2 is positive, processing in step S3 is executed. By contrast,when the determination result in step S2 is negative, processing in stepS9 is executed.

In step S3, the CPU 10 notifies the player that the remote controloperation is possible, i.e., that the control target device is operable.In this embodiment, the notification in step S3 is performed using theLEDs 34 a through 34 d of the controller 5. Specifically, the CPU 10transmits control data for lighting up the LEDs 34 a through 34 d in apredetermined pattern to the controller 5. Upon receiving the controldata, the microcomputer 42 of the controller 5 lights up the LEDs 34 athrough 34 d in the pattern in accordance with the control data. In thisembodiment, the operation of lighting up the LEDs 34 a through 34 d inthe predetermined pattern is started in step S3 and is continued untilthe remote control operation becomes impossible (step S8). In anotherembodiment, the operation of lighting up the LEDs 34 a through 34 d inthe predetermined pattern is started in step S3 and may be terminatedafter a predetermined time duration.

FIG. 18A, FIG. 18B and FIG. 18C show exemplary patterns of lighting upthe LEDs 34 a through 34 d. As shown in FIG. 18A, at least one of theLEDs 34 a through 34 d (in FIG. 18A, the LED 34 a) may be blinked at apredetermined interval. As shown in FIG. 18B, the LEDs 34 a through 34 dmay be sequentially lit up at a predetermined interval. As shown in FIG.18C, the number of LEDs among the LEDs 34 a through 34 d to be lit upmay be increased one by one, and after all the LEDs 34 a through 34 dare lit up, one of the LEDs may be lit up again. The pattern of lightingup the LEDs in order to notify that the remote control operation ispossible may be any pattern as long as the pattern is different from thepattern used when the remote control operation is impossible.

In another embodiment, the CPU 10 may notify that the remote controloperation is possible in step S3 by outputting a predetermined soundfrom the speaker 49 of the controller 5 instead of lighting up the LEDs34 a through 34 d.

By the processing in step S3, the processing mode is set to a deviceoperation mode. In the device operation mode, the control target deviceis operable using the controller 5. After step S3, processing in step S4is executed.

In step S4, the CPU 10 resets the value of a timer which is used todetermine whether or not to terminate the device operation mode. Namely,the CPU 10 sets the value of the timer to “0”. Next in step S5, thevalue of the timer is incremented. In this embodiment, the processing instep S5 is executed every frame (e.g., at an interval of 1/60 sec.), andthe value of the timer is incremented by one frame each time theprocessing in step S5 is executed.

Next in step S6, the CPU 10 determines whether or not any of theoperation buttons 32 a through 32 i of the controller 5 has beenpressed. Specifically, the CPU 10 obtains the operation data 62 from thecontroller 5 and stores the obtained operation data 62 on the mainmemory 13. The operation data 62 includes the operation button data 622.The CPU 10 refers to the operation button data 622 to determine whetheror not any of the operation buttons 32 a through 32 i of the controller5 has been pressed. When the determination result in step S6 ispositive, processing in step S10 is executed as described later. Bycontrast, when the determination result in step S6 is negative,processing in step S7 is executed.

In step S7, the CPU 10 determines whether or not a predetermined timeduration has passed since the last time when any of the operationbuttons 32 a through 32 i of the controller 5 was pressed. Herein, thevalue of the timer represents the time duration since the last time whenany of the operation buttons 32 a through 32 i of the controller 5 waspressed (in the case where no button has been pressed, the time durationsince the processing in step S4 was first executed). Thus, thedetermination in step S7 can be made by referring to the value of thetimer. When the value of the timer is larger than the predetermined timeduration, the determination result in step S7 is positive; whereas whenthe value of the timer is smaller than the predetermined time duration,the determination result in step S7 is negative. When the determinationresult in step S7 is positive, processing in step S8 is executed. Bycontrast, when the determination result in step S7 is negative, theprocessing returns to step S5. Namely, the CPU 10 repeats the processingin steps S5 through S7 from the time when the timer is reset in step S4until any of the operation buttons 32 a through 32 i is pressed or untilthe predetermined time duration passes.

In step S8, the CPU 10 terminates the notification started in step S3.Specifically, the CPU 10 transmits control data for stopping lighting upthe LEDs 34 a through 34 d in the predetermined pattern to thecontroller 5. Upon receiving the control data, the microcomputer 42 ofthe controller 5 stops lighting up the LEDs 34 a through 34 d in thepredetermined pattern. Then, the microcomputer 42 lights up the LEDs 34a through 34 d so as to show that the game apparatus 3 is not in thedevice operation mode. Specifically, the microcomputer 42 lights up onlythe LED corresponding to the assigned number among the LEDs 34 a through34 d. After step S8, processing in step S9 is executed.

In step S9, the CPU 10 executes the game program 61 stored on theoptical disc 4 to execute the game processing. In the game processing,for example, the CPU 10 causes the TV 2 to display a virtual game space,and causes a character appearing in the virtual game space to perform amotion in accordance with the operation data from the controller 5.Specific examples of the game processing will be described later. Afterstep S9, the CPU terminates the processing shown in FIG. 16.

In step S10, the CPU 10 determines whether or not a remote controloperation button among the operation buttons 32 a through 32 i of thecontroller 5 has been pressed, based on the operation data obtained instep S6. Namely, the CPU 10 determines whether or not the operation datarepresents an operation of pressing a remote control operation button.The “remote control operation button” is an operation button assigned afunction for performing a remote control operation (operation of thecontrol target device). The remote control operation button can bespecified by referring to the operation table data 634 stored on themain memory 13. For example, in the case of the operation table data 634shown in FIG. 15, the A button 32 d and the plus button 32 g are remotecontrol operation buttons. When the determination result in step S10 ispositive, processing in step S11 is executed. By contrast, when thedetermination result in step S10 is negative, processing in step S13 isexecuted.

In step S11 and S12, an infrared signal for operating the control targetdevice (TV 2) is output. First in step S11, the CPU 10 selects a signalpattern corresponding to the pressed remote control operation button.The signal pattern is selected by referring to the operation table data634. Namely, the CPU 10 first specifies an instruction associated withthe operation represented by the operation data 62 obtained in step S6in the operation table data 634. Next, the CPU 10 refers to the signaltable data 631 to select the signal pattern corresponding to thespecified instruction. When the signal table data 631 is referred to,the correspondence information regarding the device represented by thecontrol device data 633 is referred to.

Next in step S12, the CPU 10 causes the infrared LEDs 6Ra and 6Laincluded in the sensor bar 6 to output an infrared signal of the signalpattern selected in step S11. In this way, a remote control signal forcontrolling the control target device (TV 2 in this example) is output.In general, a PPM (Pulse Position Modulation) signal is output as theremote control signal. After step S12, the processing returns to stepS4.

In step S13, the CPU 10 determines whether or not the power button 32 hhas been pressed, based on the operation data 62 obtained in step S6.When the determination result in step S13 is positive, processing instep S14 is executed. By contrast, when the determination result in stepS13 is negative, the processing in step S8 is executed. When thedetermination result in step S13 is negative, an operation which isneither a remote control operation button nor the power button 32 h ispressed. Namely, in this embodiment, when an operation button which isneither a remote control operation button nor the power button 32 h ispressed, the device operation mode is terminated (step S8) and the gameprocessing is executed (step S9).

In step S14, the CPU 10 turns off the game apparatus 3 (places the gameapparatus 3 into the stand-by state), and terminates the processing inFIG. 16. The controller 5 having the power button 32 h pressed is turnedoff. In the case where a plurality of controllers 5 are communicablewith the game apparatus 3, the CPU 10 may instruct to turn off thecontrollers 5 other than the controller 5 having the power button 32 hpressed.

As described above, in this embodiment, when the game apparatus 3 isturned on, the processing mode is set to the device operation mode (stepS3). Under the condition that no input is made on any of the operationbuttons of the controller 5 for a predetermine time duration (YES instep S7) or an operation button other than the remote control operationbuttons (NO in step S10), the processing mode is switched to a gameprocessing mode (step S9). In the game processing mode, the gameprocessing is executed based on the operation made on the controller 5.

FIG. 19 shows how the TV 2 is controlled by the processing shown in FIG.16 executed by the game apparatus 3. In this embodiment, when the gameapparatus 3 is turned on, the game apparatus 3 is placed into the deviceoperation mode, in which the TV 2 as the control target device isoperable. When the player presses a button for turning on the TV 2 (theA button 32 d of the controller 5) in this state, the operation data 62indicating that such a button has been pressed is transmitted from thecontroller 5 to the game apparatus 3 (arrow (1) in FIG. 19). In thisway, the determination result in step S10 (FIG. 16) is made positive.Thus, the game apparatus 3 instructs the sensor bar 6 to output aninfrared signal for turning on the TV 2 in step S11 (arrow (2) in FIG.19). In accordance with the instruction, the markers 6R and 6L of thesensor bar 6 radiate the infrared signal (arrows (3) in FIG. 19). Theradiated infrared signal is received by the infrared receiving section52 of the TV 2. The infrared light from the markers 6R and 6L is eitherdirectly received by the infrared receiving section 52, or is reflectedby an object in the vicinity of the TV 2 (an object in the room, thewall or glass window of the room, etc.). In the latter case, thereflected light is received by the infrared receiving section 52. By theinfrared receiving section 52 receiving the infrared signal, the TV 2 isturned on. As described above, in this embodiment, the player canoperate the TV 2 using the controller 5 before starting the game. Evenif the TV 2 is off before starting the game, the player does not need tolook for the remote control of the TV 2 and can start the game only byoperating the controller 5.

Next, the game processing in step S9 will be described in detail. FIG.20 is a flowchart illustrating the game processing (step S9) in FIG. 16in detail. As described above, the game processing shown in FIG. 20 isfor a race game. During the game processing mode, the control targetdevice is the remote control car 71, not the TV 2. Specifically, whenstarting the game processing, the CPU 10 changes the contents of thecontrol target data 633 and the operation table data 634 stored on themain memory 13. More specifically, the CPU 10 stores data representingthe remote control car 71 on the main memory 13 as the control targetdata 633 in place of the data representing the TV 2. The CPU 10 alsochanges the contents of the operation table data 634 into contents foroperating the remote control car 71.

In the game processing mode, the CPU 10 continuously causes the markers6R and 6L to output the infrared light except for when an infraredsignal is output (in the game processing shown in FIG. 20, steps S45(FIG. 21) and S58 (FIG. 24)).

The game processing shown in FIG. 20 is executed as follows. First instep S31, the CPU 10 executes course creation processing. By the coursecreation processing, a race course in the virtual game space is createdbased on an instruction made to the remote control car 71 as the controltarget device. Hereinafter, the course creation processing will bedescribed with reference to FIG. 21 in detail.

FIG. 21 is a flowchart illustrating the course creation processing (stepS31) shown in FIG. 20 in detail. The course creation processing isexecuted as follows. First in step S41, the CPU 10 obtains operationdata 62 from the controller 5. The controller 5 transmits the operationdata 62 at a predetermined time interval (for example, at an interval ofequal to or shorter than one frame). The CPU 10 stores the transmittedoperation data 62 on the main memory 13. The operation data 62 includesthe marker coordinate set data 621, the operation button data 622, andthe acceleration data 623. The CPU 10 updates and stores the data 621through 623 included in the operation data 62 on the main memory 13. Inthis embodiment, the processing loop of steps S41 through S47 isexecuted for each frame.

Next in step S42, the CPU 10 determines an instruction to be given tothe remote control car 71 based on the operation made on the controller5. The instruction to be given to the remote control car 71 isdetermined by referring to the operation table data 634. Namely, the CPU10 determines the instruction, associated with the operation representedby the operation data 62 obtained in step S41 in the operation tabledata 634, as the instruction to be given to the remote control car 71.

This will be described with reference to the operation table data 634shown in FIG. 15. The CPU 10 determines whether or not the A button 32 dhas been pressed based on the operation button data 622 included in theoperation data 62. When the A button 32 d has been pressed, the CPU 10refers to the operation table data 634 to determine the instructioncorresponding to the operation of pressing the A button 32 d, i.e., theinstruction to turning on the accelerator, as the instruction to begiven to the remote control car 71. The CPU 10 also calculates therotation angle of the controller 5 around an axis in the longitudinaldirection thereof, based on the marker coordinate set data 621. The CPU10 determines the angle of the steering wheel in accordance with thecalculated rotation angle, referring to the rotation table data 634. Asdescribed above, the operation on the controller 5 for controlling theremote control car 71 is not limited to pressing the operation buttons32 a through 32 i of the controller 5, and may be rotating thecontroller 5 around an axis in the longitudinal direction thereof.Alternatively, the operation may be changing the indicated position bythe controller 5. In this embodiment, the acceleration data 623 includedin the operation data 62 in not used for the game processing. In anotherembodiment, the game processing may be executed based on theacceleration data 623.

In step S43, the CPU 10 stores the input history data 635. In thisembodiment, data representing the instruction determined in step S42 isstored on the main memory 13 as the input history data 635. Namely, dataindicating whether the accelerator of the remote control car 71 is to beon or off, and data representing the angle (rotation amount) of thesteering wheel to be given to the remote control car 71, are stored onthe main memory 13. In this embodiment, while the processing loop ofsteps S41 through S47 is being executed, the input history data 635 isadditionally stored each time step S43 is performed. On the main memory13, data representing all the instructions determined in step S42 whilethe processing loop is being executed is stored in the order of beingcreated. In this way, the history of inputs which were made during theprocessing loop of steps S41 through S46 is stored.

Next in step S44, the CPU 10 selects the signal pattern of the infraredsignal corresponding to the instruction determined in step S42. Thesignal pattern is selected by referring to the signal table data 631stored on the main memory 13. The CPU 10 selects the signal patternassociated with the instruction determined in step S42 in thecorrespondence information regarding the remote control car 71 in thesignal table data 631. Next in step S45, the CPU 10 causes the infraredLEDs 6Ra and 6La included in the sensor bar 6 to output an infraredsignal of the signal pattern selected in step S44.

In step S46, the CPU 10 generates an image representing the instructiondetermined in step S42 and causes the TV 2 to display the generatedimage. FIG. 22 shows an exemplary game image displayed on the TV 2 inthe course creation processing. As shown in FIG. 22, the display screenof the TV 2 displays a steering wheel image 66 and an image 67representing whether the accelerator is on or off as the imagerepresenting the instruction. The steering wheel image 66 is displayedas rotating in accordance with the angle of the steering wheel. When theaccelerator is set to “on”, the image 67 is displayed as “ACCELERATOR:ON” as shown in FIG. 22. When the accelerator is set to “off”, the image67 is displayed as “ACCELERATOR: OFF”. Since the image representing theinstruction is displayed, the player can visually check the operationparticulars on the remote control car 71. Thus, the player can easilyoperate the remote control car 71.

Next in step S47, the CPU 10 determines whether or not to terminate theoperation on the control target device (remote control car 71). Thisdetermination is made based on, for example, whether or not apredetermined time duration has passed since the course creationprocessing was started, or whether or not the player performed apredetermined operation of terminating the operation. When thedetermination in step S47 is positive, the processing in step S48 isexecuted. By contrast, when the determination in step S47 is negative,the processing returns to step S41.

FIG. 23 shows how the remote control car 71 is operated using thecontroller 5. The operation on the remote control car 71 using thecontroller 5 is performed by repeating the processing in steps S41through S47. Specifically, the imaging information calculation section35 of the controller 5 takes an image of the markers 6R and 6L (arrows(1) in FIG. 23), and the controller 5 transmits the imaging information(information on the marker coordinate sets) to the game apparatus 3 asthe operation data 62 (arrow (2) in FIG. 23). The game apparatus 3obtains the operation data 62 in step S41, and determines theinstruction to be given to the remote control car 71 based on theoperation data 62 in step S42. The game apparatus 3 instructs the sensorbar 6 to output an infrared signal in accordance with the determinedinstruction in steps S44 and S45 (arrow (3) in FIG. 23). Following thisinstruction from the game apparatus 3, the markers 6R and 6L of thesensor bar 6 radiate the infrared signal (arrows (4) in FIG. 23). Theradiated infrared signal is received by an infrared receiving section ofthe remote control car 71. As a result, the motion of the remote controlcar 71 is controlled. In this way, the player can operate the remotecontrol car 71 using the controller 5. In this embodiment, theinstructions made while the remote control car 71 is being operated arestored in the game apparatus 3 by the processing in step S43.

Returning to FIG. 21, in step S48, the CPU 10 creates a race course tobe constructed in the game space based on the input history data 635stored on the main memory 13. The race course may be created, forexample, as follows. The running track of the remote control car 71 iscalculated based on the input history data 635. Then, the race course iscreated so as to follow the calculated track. Data representing thecreated race course is stored on the main memory 13 as the race coursedata 636. After step S48, the CPU 10 terminates the course creationprocessing.

Returning to FIG. 20, in step S32, the CPU 10 executes race gameprocessing. By the race game processing, a character (racing car) ismoved on the race course in the virtual game space in accordance withthe operation made on the controller 5. With reference to FIG. 24, therace game processing will be described in detail.

FIG. 24 is a flowchart illustrating the race game processing (step S32)shown in FIG. 20 in detail. The race game processing is executed asfollows. First, a race course is constructed in the game space inaccordance with the race course data 636 stored on the main memory 13,and then a game image representing a part or the entirety of the gamespace and a character appearing in the game space is displayed on the TV2. After the game image is displayed, processing in step S51 isexecuted.

In step S51, the CPU 10 obtains operation data 62 from the controller 5.The processing in step S51 is substantially the same as that of stepS41. Next in step S52, the CPU 10 determines a motion of the racing carbased on the operation made on the controller 5. The motion of theracing car is determined based on data 621 through 623 included in theoperation data 62 obtained in step S51. For example, the motion of theracing car may be determined in substantially the same manner as themotion of the remote control car 71. Namely, when the operation data 62indicates that the A button 32 d has been pressed, the racing car iscaused to perform an acceleration motion, which corresponds to themotion of turning on the accelerator of the remote control car 71. Likein step S42, the angle of the steering wheel of the racing car iscalculated based on the marker coordinate set data 621, and the racingcar is caused to perform a motion of changing the proceeding directionby the angle corresponding to the angle of the steering wheel.

Next in step S53, the CPU 10 stores input history data. In thisembodiment, two types of data, i.e., data indicating whether or not theracing car has performed the acceleration motion, and data representingthe angle of the steering wheel, are stored on the main memory 13 as theinput history data 635. In this embodiment, while the processing loop ofsteps S51 through S55 is being executed, the input history data 635 isadditionally stored each time step S53 is performed. Namely, the twotypes of data generated during the processing loop are all stored in theorder of being generated. In this way, the history of inputs made whilethe processing loop of steps S51 through S55 is executed is stored.

Next in step S54, the CPU 10 causes the TV 2 to display a game imagerepresenting the motion of the racing car determined in step S52. Byexecuting the processing in step S54 for each frame, the motion of theracing car moving on the race course in the game space is displayed onthe TV 2.

Next in step S55, the CPU 10 determines whether or not to terminate therace game. The determination in step S55 is made based on, for example,whether or not the racing car operated by the player has reached thegoal or whether or not a predetermined time duration has passed sincethe race game was started. When the determination result in step S55 ispositive, processing in steps P56 through S58 is executed. By contrast,when the determination result in step S55 is negative, the processingreturns to step S51.

In steps S56 and S57, the remote control car 71 is controlled based onthe operation made on the racing car during the race game in steps S51through S55. In step S56, the CPU 10 determines the instruction to begiven to the remote control car 71 based on the input history data 635stored on the main memory 13. As described above, in this embodiment,the two types of data, i.e., data indicating whether or not the racingcar has performed the acceleration motion, and data representing theangle of the steering wheel, are stored on the main memory 13 as theinput history data 635. Thus, the two types of data are used as they areas data representing the instruction. In another embodiment, datarepresenting the history of the positions and speeds of the racing carmay be stored as the input history data, and data representing theinstruction may be calculated based on the input history data. The inputhistory data 635 includes a plurality of pieces of data representing aninput for each frame. Therefore, in step S56, the instruction isdetermined in correspondence with each piece of data. Namely, in stepS56, a series of instructions are determined in correspondence with aseries of inputs made during the race game processing.

In step S57, the CPU 10 selects signal patterns of the infrared signalscorresponding to the instructions determined in step S56. Since theseries of instructions are determined in step S56, a plurality of signalpatterns are selected in correspondence with the series of instructionsin step S57. The processing of selecting a signal pattern correspondingto each instruction is substantially the same as that of step S44.

In step S58, the CPU 10 causes the infrared LEDs 6Ra and 6La included inthe sensor bar 6 to output infrared signals of the signal patternsselected in the steps S57. Since a plurality of signal patterns areselected in step S57, a plurality of infrared signals correspondingthereto are output one by one at a predetermined interval. After stepS58, the CPU 10 terminates the race game processing shown in FIG. 24.Thus, the game processing shown in FIG. 20 is terminated.

As described above, according to the game processing shown in FIG. 20,the player can create a race course in the game space by operating theremote control car 71 before starting the race game. Owing to this, themotion of the remote control car 71 in the real world and the gamecourse in the virtual game world can be associated with each other.Thus, a game having a new type of entertaining factor by which themotion of the control target device in the real world is reflected onthe game can be provided.

According to the game processing shown in FIG. 20, the game processingis executed after the remote control car 71 is operated. Alternatively,the operation on the remote control car 71 and the game processing maybe performed at the same time. Namely, the game apparatus 3 may controlthe remote control car 71 and the racing car at the same time byoperating the controller 5. Hereinafter, game processing when the remotecontrol car 71 and the racing car are controlled at the same time willbe described with reference to FIG. 25.

FIG. 25 is a flowchart illustrating another example of the gameprocessing (step S9) shown in FIG. 16 in detail. In FIG. 25, identicalsteps as those of FIG. 21 bear identical step numbers thereto anddetailed descriptions thereof will be omitted. In the game processingshown in FIG. 25, like in the processing shown in FIG. 24, a race courseis constructed in the game space in accordance with the race course data636 stored on the main memory 13, and a game image representing a partor the entirety of the game space and the character appearing in thegame space is displayed on the TV 2. After the game image is displayed,the processing shown in FIG. 25 is executed.

As shown in FIG. 25, in step S41, operation data 62 is obtained from thecontroller 5. Next in step S42, the instruction to be given to theremote control car 71 is determined based on the operation made on thecontroller 5. In the processing shown in FIG. 25, the processing in stepS43 is not executed.

After step S42, the processing in step S44 is executed. In step S44, asignal pattern of the infrared signal corresponding to the instructiondetermined in step S42 is selected. Next in step S45, an infrared signalof the signal pattern selected in step S44 is output by the LEDs 6Ra and6La included in the sensor bar 6.

After step S45, processing in step S61 is executed. In step S61, the CPU10 determines the motion of the racing car based on the operation madeon the controller 5. The processing in step S61 is substantially thesame as that of step S52 shown in FIG. 24. In step S61, the motion ofthe character may be determined based on the instruction determined instep S42, instead of the operation made on the controller 5, i.e., theoperation data 62 stored on the main memory 13.

In FIG. 25, an infrared signal (remote control signal) of apredetermined pattern is output in step S45 from the markers 6R and 6L.Namely, the markers 6R and 6L are lit up and turned off repeatedly. Itis assumed here that the imaging element 40 of the controller 5 takes animage of the markers 6R and 6L while the remote control signal is beingoutput. The interval of the markers 6R and 6L being lit up and turnedoff as the remote control signal is generally very short, and so theimaging element 40 does not sense that the markers 6R and 6L are lit upand turned off. Therefore, even while the remote control signal is beingoutput from the markers 6R and 6L, the imaging element 40 is capable oftaking an image of the markers 6R and 6L. Even if the remote controlsignal is output from the markers 6R and 6L during the game processingexecuted using the taken image of the markers 6R and 6L (step S61) as inthis embodiment, the game processing is considered not to be influenced.

In this embodiment, the remote control signal is output even in the gameprocessing mode. In another embodiment, the remote control signal may beset not to be output in the game processing mode in order to obtain moreaccurate information. Namely, the game processing may be executed usingthe imaging information only while the infrared light is continuouslyoutput.

After step S61, in step S62, the CPU 10 causes the TV 2 to display agame image representing the racing car performing the motion determinedin step S61. The processing in step S62 is substantially the same asthat of step S54. Next in step S63, the CPU 10 determines whether or notto terminate the game processing. The determination in step S63 is madebased on, for example, whether or not the racing car operated by theplayer has reached the goal, or whether or not a predetermined timeduration has passed since the race game was started. When thedetermination result in step S63 is positive, the CPU 10 terminates thegame processing shown in FIG. 25. By contrast, when the determinationresult in step S63 is negative, the processing returns to step S41.

As described above, according to the game processing shown in FIG. 25,at the same time as the motion of the control target device (remotecontrol car 71) is controlled based on the operation made on thecontroller 5 (step S45), the motion of the character (racing car) in thegame space is controlled based on the operation made on the controller 5(step S61). Therefore, a game having a new type of entertaining factorby which the player can operate the control target device in the realworld at the same time as operating the character in the game space canbe provided.

In the above embodiment, the race game is used for an example of thegame processing. The present technology is applicable to any game aslong as the game processing is executed in accordance with the operationmade on the controller 5. For example, the motion of another controltarget device (e.g., TV 2) may be controlled with a command selectionscreen image during the game. Hereinafter, game processing according toa different embodiment of the present technology will be described withreference to FIG. 26 and FIG. 27.

FIG. 26 shows an exemplary command selection screen image displayed inthe game processing according to a different embodiment. As shown inFIG. 26, the screen of the TV 2 displays command images 81 through 84representing commands A through D which can be indicated by the player,an gauge image 85 and a knob image 86 representing the sound volume ofthe TV 2, and a cursor 87 for designating the image on the screen. Theplayer can designate one of the command images 81 through 84 using thecursor 87 to indicate the command corresponding to the designatedcommand image. The player can also move the knob image 86 up and down inthe gauge image 85 using the cursor 87 to indicate the sound volume. Thecursor 87 is moved by an operation of changing the indicated position bythe controller 5.

FIG. 27 is a flowchart illustrating a flow of the game processing in theembodiment shown in FIG. 26. With reference to FIG. 27, the gameprocessing when the command selection screen image is displayed at anappropriate timing during the game (for example, when the player makes apredetermined indication) will be described and the other gameprocessing irrelevant to the present technology will be omitted.

The game processing shown in FIG. 27 is executed as follows. First instep S71, the CPU 10 causes the TV 2 to display a command selectionscreen image. Next in step S72, the CPU 10 obtains operation data 62from the controller 5. The processing in step S72 is substantially thesame as that of step S41.

Next in step S73, the CPU 10 calculates the position at which the cursor87 is to be displayed, using the marker coordinate set data 621 includedin the operation data 62. Specifically, the indicated position by thecontroller 5 is calculated based on the marker coordinate sets, and theposition at which the cursor 87 is to be displayed is set at thecalculated indicated position.

Next in step S74, the CPU 10 determines whether or not the knob image 86is designated by the cursor 87. The determination in step S74 is madebased on whether or not the position at which the cursor 87 is displayedis within a predetermined distance from the area in which the knob image86 is displayed. When the determination result in step S74 is positive,processing in step S77 is executed as described later. By contrast, whenthe determination result in step S74 is negative, the processing in stepS75 is executed.

In step S75, the CPU 10 determines whether or not the player hasperformed a command selection operation. Herein, the “command selectionoperation” means an operation of pressing the A button 32 d of thecontroller 5 in the state where one of the command selection images 81through 84 is designated by the cursor 87. When the determination resultin step S75 is positive, processing in step S76 is executed. Bycontrast, when determination result in step S75 is negative, theprocessing returns to step S72.

In step S76, the CPU 10 executes game processing in accordance with thecommand selected in step S75. The specific content of the command may beanything which is related to the situation in the game or the setting ofthe game. After step S76, processing in step S86 is executed asdescribed later.

In step S77, the CPU 10 determines whether or not the operation ofselecting the knob image 86 by the cursor 87 has been performed. Herein,the “operation of selecting the knob image 86 by the cursor 87” is theoperation of pressing the A button 32 d of the controller 5 in the statewhere the knob 86 is designated by the cursor 87. In step S77, it isdetermined whether or not the A button 32 d has been pressed. When thedetermination result in step S77 is positive, processing in step S78 isexecuted. By contrast, when determination result in step S77 isnegative, the processing returns to step S72.

In step S78, the CPU 10 moves the knob image 86 in accordance with theposition of the cursor 87. Specifically, the knob image 86 is moved upand down in the gauge image 85 in accordance with the position at whichthe cursor 87 is displayed.

Next in step S79, the CPU 10 selects a signal pattern in accordance withthe position of the knob image 86 moved in step S78. Namely, the CPU 10determines the sound volume of the TV 2 to be set, in accordance withthe position of the knob image 86, and selects a signal patterncorresponding to the instruction for setting the sound volume asdetermined. Next in step S80, the CPU 10 causes the LEDs 6Ra and 6Laincluded in the sensor bar 6 to output an infrared signal of the signalpattern selected in step S79. By the infrared receiving section 52 ofthe TV 2 receiving the output infrared signal, the sound volume of theTV 2 is set.

In step S81, the CPU 10 determines whether or not the knob image 86 hasbeen released from the selection by the cursor 87. The determination instep S81 is made based on whether or not the A button 32 d has beenlifted. When the determination in step S81 is positive, processing instep S82 is executed. By contrast, when the determination in step S81 isnegative, the processing returns to step S78.

In step S82, the CPU 10 determines whether or not to terminate thedisplay of the command selection screen image. The determination in stepS82 is made based on, for example, whether or not the player has made apredetermined indication of terminating the display of the commandselection screen image. When the determination in step S82 is positive,the CPU 10 terminates the game processing shown in FIG. 27. By contrast,when the determination in step S82 is negative, the processing returnsto step S72.

By such processing, the player can press the A button 32 d in the statewhere the cursor 87 is aligned to the position of the knob image 86 tomove the knob image 86 up and down while the A button 32 d is pressed.By moving the knob image 86 by the cursor 87, the player can change thesound volume of the TV 2. In addition, the player can press the A buttonin the state where the cursor 87 is aligned to one of the command images81 through 84 to execute the command corresponding to the command image.As shown in FIG. 27, while the game processing is being executed, eitherthe game operation or the remote control operation may be selectivelyperformed.

In the above embodiment, the TV and the remote control car are thecontrol target devices. In another embodiment, the control target devicemay be any device, the motion of which is controllable by an infraredremote control signal. For example, the control target device may be ahome-use electric appliance such as an air conditioner or the likelocated in the same room as the game system, or may be a robotperforming the same motion as the character appearing in the game space.

In the above embodiment, the infrared signal which is output from eachof the markers 6R and 6L represents an instruction for causing thecontrol target device to perform a predetermined motion, i.e., is aremote control signal. In another embodiment, the infrared signal may beany signal usable in the control target device, and may represent animage or a sound to be reproduced by the control target device orrepresent parameter data usable by the control target device. Even whenthe infrared signal represents only an image or a sound, when thecontrol target device receiving the infrared signal reproduces the imageor the sound, such an infrared signal represents an instruction forcausing the control target device to perform a predetermined motion(reproduction). Even when the infrared signal represents the parameterdata, when the control target device receiving the infrared signalexecutes predetermined processing using the parameter data, such aninfrared signal represents an instruction for causing the control targetdevice to perform a predetermined motion (processing). Hereinafter, withreference to FIG. 28, a modification of the embodiment described abovewith reference to FIG. 1 through FIG. 24 will be described.

FIG. 28 shows the modification. In FIG. 28, speakers 72 a and 72 b arethe control target devices. The speakers 72 a and 72 b each include aninfrared receiving section, and receives an audio signal transmitted asan infrared signal and outputs the received audio signal as a sound. Inthe modification shown in FIG. 28, storage means of the game apparatus 3(the main memory 13, the flash memory 18, or the like) stores at leastone signal pattern representing an audio signal to be reproduced throughthe speakers 72 a and 72 b. Like in the above embodiment, the controller5 takes an image of the markers 6R and 6L (arrows (1) in FIG. 28), andtransmits operation data including information on the taken image(specifically, information on the marker coordinate sets) to the gameapparatus 3 (arrow (2) in FIG. 28). Based on the received operationdata, the game apparatus 3 specifies an operation made on the controller5. When an operation for causing the speakers 72 a and 72 b (or eitherthe speaker 72 a or 72 b) to output a sound is performed on thecontroller 5, the game apparatus 3 determines the signal patternrepresenting the audio signal to be reproduced, based on the operation.The game apparatus 3 outputs the determined signal pattern to themarkers 6R and 6L (arrow (3) in FIG. 28). In this way, the infraredsignal as the audio signal is radiated from the markers 6R and 6L, andthe speakers 72 a and 72 b receive the infrared signal (arrows (4) inFIG. 28). As a result, the sound represented by the received infraredsignal is reproduced through the speakers 72 a and 72 b. Thus, the gameapparatus 3 can cause the control target device to reproduce a sound inaccordance with the operation made on the controller 5. In FIG. 28, theaudio signal to be reproduced by the control target device is output asan infrared signal. Alternatively, an image signal may be output as aninfrared signal from a display such as a monitor or the like used as thecontrol target device.

One example of the game using the structure shown in FIG. 28 is ashooting game. In such a shooting game, the player presses apredetermined shooting button (for example, the A button 32 d) providedin the controller 5. Then, the game apparatus 3 radiates an infraredsignal of a signal pattern representing the audio data corresponding toa shooting sound from the markers 6R and 6L. Thus, the shooting sound isreproduced from the speakers 72 a and 72 b. As described above,according to this embodiment, an infrared signal as an audio signal isoutput from the markers 6R and 6L in accordance with the operation madeon the controller 5, so that the control target device (speaker) canreproduce a sound. In the shooting game, an image of the game space maybe displayed on the screen of the TV 2 and the indicated position by thecontroller 5 may be used as the aim.

The “parameter data usable by the control target device” is, forexample, data on a game parameter in the case where the control targetdevice is a game apparatus different from the gate apparatus 3. Forexample, an infrared signal, representing data on the parameter to beset for the character appearing in the game which is executed by thegame apparatus as the control target device, may output from the markers6R and 6L. At this point, the game apparatus as the control targetdevice receives the infrared signal and stores the data representing thereceived infrared signal. The stored data is used for the gameprocessing at an appropriate timing.

As described above, the present embodiments are applicable to, forexample, a game apparatus used for a game system for the purpose of, forexample, using an input device of the game apparatus to operate anotherdevice.

While the example embodiments have been described in detail, theforegoing description is in all aspects illustrative and notrestrictive. It is understood that numerous other modifications andvariations can be devised without departing from the scope of theembodiments.

1. An information processing apparatus for use with a display device,comprising: a handheld control unit having manually operable controlsthereon and having an inertial sensor therein, the inertial sensorproviding an output indicating at least aspects of orientation of thehandheld control unit for use in positioning objects on displays of thedisplay device; a processing unit operatively coupled to the handheldcontrol unit and the manually operable controls; and a radiation emitteroperatively coupled to the processing unit; the processing unitcontrolling the radiation emitter to emit a patterned radiation signalcorresponding to a command, the patterned radiation signal remotelycontrolling at least the display device, the processing unit selectingthe patterned radiation signal from a plurality of patterned radiationsignals corresponding to different display device commands.
 2. Theinformation processing apparatus of claim 1, wherein the radiationemitter is an infrared light emitter.
 3. A method of operating aninformation processing apparatus for use with a display device, theinformation processing apparatus comprising a processing unit and ahandheld control unit having manually operable controls thereon andhaving an inertial sensor therein, the inertial sensor providing anoutput indicating at least aspects of orientation of the handheldcontrol unit for use in positioning objects on displays of the displaydevice, the method comprising: (a) operatively coupling the processingunit to the handheld control unit and the manually operable controls;(b) operatively coupling a radiation emitter to the processing unit: (c)the processing unit controlling the radiation emitter to emit apatterned radiation signal corresponding to a command, the patternedradiation signal remotely controlling at least the display device, andselecting the patterned radiation signal from a plurality of patternedradiation signals corresponding to different display commands.
 4. Themethod of operating an information processing apparatus for use with adisplay device according to claim 3, wherein the radiation emitter is aninfrared light emitter.
 5. A non-transitory computer-readable physicalstorage medium having tangibly recorded thereon an informationprocessing program executable by a computer of an information processingapparatus, the information processing apparatus comprising a handheldcontrol unit having manually operable controls thereon and having aninertial sensor therein, the inertial sensor providing an outputindicating at least aspects of orientation of the handheld control unitfor use in positioning objects on displays of a display device, theinformation processing program causing the computer to execute: (a)operatively coupling the computer to the handheld control unit and themanually operable controls; (b) operatively coupling a radiation emitterto the computer: (c) the computer controlling the radiation emitter toemit a patterned radiation signal corresponding to a command, thepatterned radiation signal remotely controlling at least the displaydevice, and selecting the patterned radiation signal from a plurality ofpatterned radiation signals corresponding to different display commands.6. The non-transitory computer-readable physical storage mediumaccording to claim 5, wherein the radiation emitter is an infrared lightemitter.
 7. An information processing system comprising: a displaydevice; a handheld control unit having manually operable controlsthereon and having an inertial sensor therein, the inertial sensorproviding an output indicating at least aspects of orientation of thehandheld control unit for use in positioning objects on displays of thedisplay device; a processing unit operatively coupled to the handheldcontrol unit and the manually operable controls; and a radiation emitteroperatively coupled to the processing unit; the processing unitcontrolling the radiation emitter to emit a patterned radiation signalcorresponding to a command, the patterned radiation signal remotelycontrolling at least the display device, the processing unit selectingthe patterned radiation signal from a plurality of patterned radiationsignals corresponding to different display device commands.
 8. Theinformation processing system according to claim 7, wherein theradiation emitter is an infrared light emitter.